This invention relates to an interpolation processing apparatus, an interpolation processing method and an image display apparatus, and more particularly to an interpolation processing apparatus, an interpolation processing method and an image display apparatus suitably applied to format conversion of image data in which, for example, a programmable video DSP (Digital Signal Processor) is used.
Conventionally, various apparatus such as an image display apparatus execute a process for format conversion by converting the number of pixels of image data by an interpolation arithmetic operation process. In such a process for format conversion as just mentioned, a device for exclusive use in the form of an ASIC (Application Specific IC) is used popularly.
In particular, various formats are available for image data. For example, in the VGA (Video Graphics Array), a screen is composed of 640xc3x97480 pixels or dots. Meanwhile, in the SVGA (Super VGA), a screen is composed of 800xc3x97600 pixels or dots; in the XGA (extended Graphics Array), a screen is composed of 1,024xc3x97768 pixels or dots; and in the UXGA (Ultra XGA), a screen is composed of 1,600xc3x971,200 pixels or dots.
The VGA and so forth mentioned above are formats applied to computer equipments. However, for image data, various formats which are different in the number of pixels, scanning number of lines and so forth also depending upon the system of television broadcasting are available in addition to the formats applied to computer equipments.
Meanwhile, as display apparatus for displaying image data, fixed pixel displaying devices such as a liquid crystal display (LCD) device and a digital micromirror device (DMD) are available. Each of such fixed pixel displaying devices can display a fixed number of pixels and has a unique resolution based on the number of pixels.
If a display apparatus having a unique resolution is connected so that image data of a format which does not comply with the resolution are inputted thereto, then a process of format conversion for converting the number of pixels is required. This format conversion process is executed using an interpolation arithmetic operation process. More particularly, if it is intended to display a television broadcast on a liquid crystal display apparatus, then a process of format conversion is required because of difference between the format used by the liquid crystal display apparatus and the format of the television broadcast. An interpolation arithmetic operation process for such conversion of the number of pixels is applied not only to such format conversion of image data as described above but also to partial enlargement or contraction of an image.
Such an interpolation arithmetic operation process as described above is executed by successive weighted addition of image data with weighting coefficients produced using an interpolation function. In order to allow format conversion, enlargement of an image or reduction of an image with a high picture quality, various functions are selectively used as the interpolation function.
FIG. 18 is a characteristic diagram illustrating an interpolation function f(x) according to a bilinear approximate function which is one of such interpolation functions as just described. The interpolation function f(x) illustrated in FIG. 18 can be expressed in accordance with the following expressions using a phase x normalized with a pixel interval:
xe2x80x83f(x)=1xe2x88x92|x|(|x|xe2x89xa61) f(x)=0 (|x| greater than 1)xe2x80x83xe2x80x83(1)
Meanwhile, FIG. 19 is a diagram illustrating a relationship of pixels in a horizontal direction or a vertical direction between VGA image data and SVGA image data. The number of pixels of the VGA is 640xc3x97480 dots while the number of pixels of the SVGA is 800xc3x97600 dots, and the ratio in number of pixels between the VGA and the SVGA is 4:5. Consequently, the relationship of pixels illustrated in FIG. 19 repetitively appears in a horizontal direction and a vertical direction. It is to be noted that the values 5 and 4 between pixels in FIG. 19 denote numerical values representative of pixel intervals in image data of the VGA and image data of the SVGA, respectively.
The pixel of the SVGA denoted by reference character a in FIG. 19 overlaps with the pixel of the VGA denoted by reference character A and has a phase x equal to the value 0 with respect to the pixel A. Meanwhile, the phase x of the pixel a has the different value 1 with respect to the pixel B next to the pixel A and has values greater than 1 individually with respect to the pixels denoted by reference characters C to E. Consequently, when the interpolation function of the expressions (1) is used to perform format conversion of image data of the VGA into image data of the SVGA, with regard to the pixel a, a weighting coefficient of the value 1 is produced for the pixel A, and another weighting coefficient of the value 0 is produced for the other pixels B to E.
The pixel of the SVGA denoted by reference character b has the phase x of the value xe2x88x92⅘ with respect to the pixel A. Meanwhile, the phase x of the pixel b has the value ⅕ with respect to the pixel B next to the pixel A and has values greater than 1 individually with respect to the pixels C to E. Consequently, when format conversion of image data of the VGA into image data of the SVGA is to be performed, with regard to the pixel b, weighting coefficients of the values ⅕ and ⅘ are produced for the pixels A and B, respectively, and another weighting coefficient of the value 0 is produced for the other pixels C to E similarly.
The pixel of the SVGA denoted by reference character c has the phase x of the value xe2x88x92⅗ with respect to the pixel B. Meanwhile, the phase x of the pixel c has the value ⅖ with respect to the pixel C next to the pixel B and has values greater than 1 individually with respect to the other pixels A, D and E. Consequently, when format conversion of the image data is to be formed, with regard to the pixel c, weighting coeffecients of the values ⅖ and ⅗ are produced for the pixels B and C, respectively, and another weighting coeffecient of the value 0 is produced for the other pixels A, D and E similarly.
Further, when format conversion of image data is to be performed, with regard to the pixel of the SVGA denoted by reference character d, weighting coefficients of the values ⅗ and ⅖ are produced for the pixels C and D, respectively, and with regard to the pixel of the SVGA denoted by reference character e, weighting coeffecients of the values ⅘ and ⅕ are produced for the pixels D and E, respectively.
Consequently, when the bilinear approximate function is used to perform format conversion of image data of the VGA into image data of the SVGA, a weighted addition process represented by the following expressions is repeated to produce image data of the SVGA:
a=1xc3x97A
b=⅕xc3x97A+⅘xc3x97B
c=⅖xc3x97B+⅗xc3x97C
d=⅗xc3x97C+⅘xc3x97D
e=⅘xc3x97D+⅕xc3x97E
xe2x80x83f=1xc3x97Exe2x80x83xe2x80x83(2)
A device for exclusive use which employs an ASIC includes, in order for the arithmetic operation process given by the expressions (2) above to be executed at a high speed, a coefficient generation circuit for holding and successively outputting weighting coefficients, a multiplication circuit for multiplying input image data by the weighting coefficients to weight the input data, and an addition circuit for adding results of the weighting by the multiplication circuit. The device can thus process an image on the real time basis and display also moving pictures without giving an unfamiliar feeling to a viewer.
While such a device for exclusive use employing an ASIC is used, in the field of image processing, various image processes are executed using a device for universal use in the form of a DSP (Digital Signal Processor). The DSP is a processor wherein an ALU (Arithmetic Logic Unit) operates in accordance with a sequence programmed in advance to execute an aimed signal process, and can be applied to various processes by changing the program.
A DSP adapted for image processing is constructed such that a large number of ALUs are disposed so as to allow simultaneous parallel processing of successive data so that also image data having a high sampling rate can be processed. A DSP of the type just described is generally called programmable video DSP.
FIG. 20 is a block diagram showing a programmable video DSP. Referring to FIG. 20, the programmable video DSP 1 shown includes an arithmetic operation processing section 3 to which time series successive image data DV1 are inputted through a buffer 2. The arithmetic operation processing section 3 executes a predetermined arithmetic operation process for the image data DV1 and outputs resulting data through another buffer 4. The arithmetic operation processing section 3 is formed from a plurality of processor elements (PEs) 5.
Each of the PEs 5 shown includes, as shown in FIG. 21, an ALU 6 including a multiplier, and a local memory 7 for storing input and output data to and from the ALU 6. Each of the PEs 5 repetitively performs a process of multiplying and adding data stored in the local memory 7 thereof and storing resulting data into the local memory 7 to execute a predetermined arithmetic operation process in accordance with an instruction from a controller 8 (FIG. 20). Each of the PEs 5 temporarily stores image data DV1 of the buffer 2 into the local memory 7, and further stores arithmetic operation process results DV2 into the local memory 7 and outputs the arithmetic operation process results DV2 to the buffer 4. Consequently, the programmable video DSP 1 can be represented in a modeled form from the ALUs 6 and the local memories 7 which form the PEs 5 as seen in FIG. 22.
The controller 8 records a series of processing procedures into a memory 9 under the control of an external apparatus and outputs commands to the PEs 5 in accordance with the recorded processing procedures. Control of the PEs 5 by the controller 8 is performed in accordance with a single instruction-stream multiple data-stream (SIMD) system wherein the same command COM is outputted to all of the PEs 5 so that the command is executed simultaneously and parallelly. It is to be noted that a horizontal synchronizing signal HD, a vertical synchronizing signal VD and so forth are inputted to the programmable video DSP 1, and the horizontal synchronizing signal HD and the vertical synchronizing signal VD are used as a reference for operation.
More particular description is given by way of an example wherein a programmable video DSP 1 is used to form a low-pass filter according to the bilinear approximate function given as the expressions (1) hereinabove. The relationship of pixels before and after a process by a low-pass filter can be represented in such a manner as seen in FIG. 23 in contrast to FIG. 19. In this instance, as the low-pass filter is formed, the interpolation function f(x) thereof has a characteristic widened in the direction of the phase x as indicated by broken lines in FIG. 18 in accordance with a characteristic of the low-pass filter, and also the expressions (1) are applied in a correspondingly modified form.
Here, it is assumed that the tap number is 3 and weighting coefficients according to a combination of, for example, (0.5, 1.0. 0.5) are calculated using the bilinear approximate function modified in such a manner as described above. It is to be noted, when the weighting coefficients are arithmetically operated actually, filter coefficients normalized with a sum total of the coefficients are used for the arithmetic operation and the combination of coefficients of (0.25, 0.5, 0.25) is used. In this instance, as all output image data and input image data are held in an equal phase relationship as seen from the phase relationship of FIG. 23, equal filter coefficients are applied to all of the output image data.
Consequently, in an arithmetic operation process in which the weighting coefficients according to the combination of (0.5, 1.0, 0.5) are used, successive output image data can be band-limited by repetition of the same arithmetic operation process in which corresponding input image data and image data preceding and following the input image data are used as given by the following expression:
Bxe2x80x2=xc2xcxc3x97A+xc2xdxc3x97B+xc2xcxc3x97C
Cxe2x80x2=xc2xcxc3x97B+xc2xdxc3x97C+xc2xcxc3x97D
Dxe2x80x2=xc2xcxc3x97C+xc2xdxc3x97D+xc2xcxc3x97Exe2x80x83xe2x80x83(3)
In the programmable video DSP 1, the successive input image data A to H are successively stored into the local memory 7 of each of the PEs as seen in FIG. 24A, and then a command COM is issued to all of the PEs 5 to multiply the image data A to H stored in the local memory 7 by 0.5 as seen in FIG. 24B. Consequently, each of the PEs 5 multiplies the image data A to H by 0.5 and stores resulting image data into the local memory 7 as seen in FIG. 24C (the processing described is a process corresponding the second term of the right side of each of the expressions (3)).
Then, in the programmable video DSP 1, another command COM is issued to all of the PEs 5 to multiply the image data A to H stored in the directly preceding PEs 5 by 0.25 and add the products to the results of the processing stored in all of the PEs 5 as seen in FIG. 24D. Consequently, each of the PEs 5 multiplies the corresponding image data A to H by 0.25, adds the products to the image data obtained by multiplication by 0.5 in the preceding processing and stores the sums into the local memory 7 as seen in FIG. 24E (the processing described is a process corresponding to the first and second terms of the right side of each of the expressions (3)).
Further, in the programmable video DSP 1, another command COM is issued to all of the PEs 5 to multiply the image data A to H stored in the directly following PEs 5 by 0.25 and add the products to the results of the processing stored in all of the PEs 5 as seen in FIG. 24F. Consequently, each of the PEs 5 multiplies the corresponding image data A to H by 0.25, adds the products to the image data obtained by multiplication by 0.5 in the preceding processing and stores the sums into the local memory 7 as seen in FIG. 24G (the processing described is a process corresponding to the right side of each of the expressions (3)).
Consequently, the programmable video DSP 1 can process successive image data DV1 by repeating a series of processing steps of fetching the image data DV1 into the PEs 5, executing the predetermined processing, storing results of the processing into the local memories 7 of the PEs 5 and outputting the processing results when necessary.
Consequently, the programmable video DSP 1 serves as a low-pass filter for a horizontal direction if the successive input image data A to H are image data which appear successively in the horizontal direction. However, the programmable video DSP 1 serves as a low-pass filter for a vertical direction where the successive input image data A to H are image data which appear successively in the vertical direction.
In the programmable video DSP 1, where such PEs 5 are disposed for one line as seen in FIG. 25 so that image data are processed in a unit of one line, if an arithmetic operation process for one line is completed within a period of one scanning line, then the successive image data DV1 can be processed on the real time basis.
If the programmable video DSP 1 having such a high degree of universal use as described above can execute such processes as format conversion, enlargement or contraction of an image and so forth, then this eliminates the necessity for use of a device for exclusive use and can provide various advantages.
However, where a programmable video DSP of the type described wherein all PEs are controlled simultaneously and parallelly with a single command is applied to processes such as format conversion, enlargement or contraction of an image and so forth, there is a problem in that the processing is complicated and consequently image data cannot be processed on the real time basis.
For example, where the interpolation arithmetic operation processing according to the expressions (2) given hereinabove with reference to FIG. 19 is executed, since different arithmetic operation processes are applied to the output image data a, b, c, . . . , the programmable video DSP 1 successively performs the arithmetic operations for the output image data a, b, c, . . . as seen in FIGS. 26A to 26X.
In particular, in the programmable video DSP 1, the input image data A, B, C, . . . for one line are stored into the local memory 7 as seen in FIG. 26A, and dummy data is interposed at a predetermined interval so as to make the number of the input image data equal to the number of pixels of the output image data a, b, c, . . . and stores the input image data A, B, C, . . . for one line into the local memory 7.
Then, in the programmable video DSP 1, a command COM is issued to mask the other PEs 5 than the PE 5 corresponding to the output image data a and multiply the input image data A stored in the local memory 7 only of the PE 5 corresponding to the output image data a by the value 1 as seen in!FIG. 26B. Consequently, the image data A corresponding to the input image data a is multiplied by 1 and stored into the local memory 7 as seen in FIG. 26C (this is a process corresponding to the first expression of the expressions (2)). The programmable video DSP 1 thereby completes the arithmetic operation process for the output image data a.
After the arithmetic operation process for the output image data a is completed in this manner, the programmable video DSP 1 starts an arithmetic operation process for the output image data b. In particular, the programmable video DSP 1 issues a command COM to mask the PEs 5 other than the PE 5 corresponding to the output image data b and multiply the input image data B stored in the local memory 7 only of the PE 5 by the value 0.8 as seen in FIG. 26D. Consequently, the image data B corresponding to the input image data b is multiplied by 0.8 and resulting data is stored into the local memory 7 as seen in FIG. 26E (this is a process corresponding to the second term of the right side of the second expression of the expressions (2)).
Then, the programmable video DSP 1 similarly issues a command COM to mask the PEs 5 other than the PE 5 corresponding to the output image data b, multiply the input image data A stored in the PE 5 immediately preceding to the PE 5 by the value 0.2 and adds the product to the result of the processing stored in the PE 5 as seen in FIG. 26F. Consequently, the immediately preceding image data A is multiplied by 0.2 and resulting data is added to the image data 0.8B obtained and stored in the preceding process, and then the sum is stored into the local memory 7 as seen in FIG. 26G (this is a process corresponding to the right side of the second expression of the expressions (2)). The programmable video DSP 1 thereby completes the arithmetic operation process for the output image data B.
The programmable video DSP 1 successively executes such processes as described above for the output image data a, b, c, . . . as seen in FIGS. 26H and 26I. When the arithmetic operation process for all of the output image data a, b, c, . . . is completed, results of the arithmetic operation process for one line are stored in the local memory 7 as seen in FIG. 26X. Consequently, where the programmable video DSP 1 is applied to an interpolation arithmetic operation process, it is necessary to repeat much complicated processing, and much time is required for the arithmetic operation processing and real time processing cannot be anticipated. Further, also it cannot be avoided that the memory 9 which describes such a processing procedure as described above has a large capacity, and also the preparation operation for the program itself is complicated.
It is to be noted that, as described hereinabove with reference to FIG. 19, since the phase relationship between the input image data A, B, C, . . . and the output image data a, b, c, . . . is repeated in a predetermined period in such format conversion as described above, it is a possible idea to simplify the arithmetic operation process by modifying the setting of the mask so that, upon the arithmetic operation process for the output image data a, also the image data f which has the same phase relationship to the input image data as the output image data a is processed simultaneously. Even this simplified arithmetic operation processing, however, cannot solve the problem described above completely because the arithmetic operation process must be repeated for the successive:output image data after all. Further, in a process for enlargement or contraction at a fixed ratio, the periodical relationship between such input image data A, B, C, . . . and output image data a, b, c, . . . sometimes becomes a long period, and also the interpolation arithmetic operation process itself sometimes becomes complicated. In such an instance, further increased time is required.
Also it is a possible idea to use such a programmable video DSP 1 as described above such that image data which are adjacent in the vertical direction are stored into the PEs 5 and a weighted addition process between the image data adjacent in the vertical direction is performed to effect an interpolation arithmetic operation process in the vertical direction. Where the scheme just described above is employed, it is possible for the single programmable video DSP 1 to perform an interpolation arithmetic operation process for both of the horizontal direction and the vertical direction by allocating results of addition in the horizontal direction as the image data adjacent each other in the vertical direction which are to be stored into the PEs 5. It is to be noted that the single programmable video DSP 1 can execute an interpolation arithmetic operation process for the horizontal direction and the vertical direction similarly even if an interpolation arithmetic operation process for the vertical direction is performed first and then an interpolation arithmetic operation process for the horizontal direction is performed.
However, in such an interpolation arithmetic operation process for the vertical direction as described above, although the PEs use the same weighting coefficients to perform weighted arithmetic operation, since the weighting coefficients are different between different lines, different commands COM must be issued to the different lines. Consequently, it cannot be avoided to repeat complicated processing also for the vertical direction.
It is an object of the present invention to provide an interpolation processing apparatus, an interpolation processing method and an image display apparatus wherein processes such as format conversion can be executed by a comparatively simple process by a digital signal processor of the type wherein operations of all processor elements are controlled with one command.
In order to attain the object described above, according to the present invention, in a construction wherein operations of a plurality of element processors are controlled simultaneously and parallelly, the element processors are controlled so that they produce corresponding weighting coefficients based on number data allocated thereto thereby to allow such processes as format conversion to be executed by simpler processing than ever. Further, an interpolation process is performed using an arithmetic operation expression common to different lines wherein position information of the number of lines is used as a variable.
More specifically, according to an aspect of the present invention, there is provided an interpolation processing apparatus, comprising a plurality of element processors each for performing an arithmetic operation process of input data successively inputted thereto, and control means for commonly controlling the element processors so that the element processors individually perform a weighted addition process of the input data with predetermined weighting coefficients to effect interpolation processing for the input data, each of the element processors having a memory for temporarily storing at least data in the course of the arithmetic operation process, the memories having number data allocated thereto which individually correspond to sequential order numbers of the input data, the control means controlling the element processors to perform an arithmetic operation process based on the number data to produce the weighting coefficients which are to be used for the weighted addition process by the corresponding element processors.
According to another aspect of the present invention, there is provided an interpolation processing method wherein a plurality of element processors each for performing an arithmetic operation process of input data successively inputted thereto are commonly controlled by control means so that the element processors individually perform a weighted addition process of the input data with predetermined weighting coefficients to effect interpolation processing for the input data, the interpolation processing method controlling the control means to control the element processors to allocate, to memories individually provided in the element processors for temporarily storing at least data in the course of the arithmetic operation process, number data which individually correspond to sequential order numbers of the input data, execute an arithmetic operation process based on the allocated number data to produce the corresponding weighting coefficients, and execute a process of weighted addition with the produced weighting coefficients.
According to a further aspect of the present invention, there is provided an interpolation display apparatus, comprising a plurality of element processors each for performing an arithmetic operation process of input image data successively inputted thereto, control means for commonly controlling the element processors so that the element processors individually perform a weighted addition process of the input image data with predetermined weighting coefficients to effect interpolation processing for the input data to produce output image data, and a display screen for displaying the output image data of the control means, each of the element processors having a memory for temporarily storing at least data in the course of the arithmetic operation process, the memories having number data allocated thereto which individually correspond to sequential order numbers of the input image data, the control means controlling the element processors to perform an arithmetic operation process based on the number data to produce the weighting coefficients which are to be used for the weighted addition process by the corresponding element processors.
With the interpolation processing apparatus, interpolation processing method and image display apparatus, if the element processors produce individually corresponding weighting coefficients based on the number data allocated thereto, then it is possible to output the same command to all of the element processors so that the element processors may produce corresponding weighting coefficients simultaneously and parallelly. Consequently, such repetitive outputting of commands for the individual element processors as in the prior art can be prevented, and weighting coefficients can be set to the individual element processors with a command common to all of the element processors. Further, the same command can be issued to all of the element processors so that arithmetic operations of weighted addition can be executed simultaneously and parallelly with the weighting coefficients held by the element processors. Consequently, interpolation results by the weighted addition can be obtained by simple processing.
According to a still further aspect of the present invention, there is provided an interpolation processing apparatus which repetitively executes a first interpolation process in a horizontal direction or a vertical direction and a second interpolation process in a direction perpendicular to the direction of the first interpolation process to process image data, comprising a plurality of element processors each for performing an arithmetic operation process of successive ones of the image data, and control means for commonly controlling the element processors, each of the element processors having a memory for temporarily storing at least data in the course of the arithmetic operation process, each of the memories storing a plurality of ones of the image data which are adjacent in a horizontal direction or a vertical direction to each other, the control means controlling the element processors such that, in the first or second interpolation process, each of the element processors uses position information of that one of the image data which relates to an interpolation process as a variable to perform the interpolation arithmetic operation using an arithmetic operation expression same as those used by the other element processors.
According to a yet further aspect of the present invention, there is provided an interpolation processing method wherein a plurality of element processors each for performing an arithmetic operation process of input data successively inputted thereto are commonly controlled by control means so that a first interpolation process in a horizontal direction or a vertical direction and a second interpolation process in a direction perpendicular to the direction of the first interpolation process are repetitively performed to process the image data, the interpolation processing method controlling the control means to control the element processors to store a plurality of ones of the image data which are adjacent in a horizontal direction or a vertical direction to each other into a memory of each of the element processors, and use, in the first or second interpolation process, position information of that one of the image data which relates to an interpolation process as a variable to perform interpolation arithmetic operation using an arithmetic operation expression same as those used by the other element processors.
According to a yet further aspect of the present invention, there is provided an image display apparatus which repetitively executes a first interpolation process in a horizontal direction or a vertical direction and a second interpolation process in a direction perpendicular to the direction of the first interpolation process to process input image data to produce output image data and outputs the output image data to a display screen, comprising a plurality of element processors each for performing an arithmetic operation process of successive ones of the input image data, and control means for commonly controlling the element processors, each of the element processors having a memory for temporarily storing at least data in the course of the arithmetic operation process, each of the memories storing a plurality of ones of the image data which are adjacent in a horizontal direction or a vertical direction to each other, the control means controlling the element processors such that, in the first or second interpolation process, each of the element processors uses position information of that one of the image data which relates to an interpolation process as a variable to perform the interpolation arithmetic operation using an arithmetic operation expression same as those used by the other element processors.
With the interpolation processing apparatus, interpolation processing method and image display apparatus, the element processors are controlled so that they may perform interpolation arithmetic operations in accordance with the same arithmetic operation expression among the interpolation processes using the position information of image data which relates to an interpolation process as a variable. Consequently, where the interpolation process is, for example, an interpolation process for the vertical direction, interpolation process results can be obtained by the control common to the individual lines. Consequently, interpolation results can be obtained by simpler processing than ever.